{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Part 3 - Introduction to Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/e/ed/Pandas_logo.svg/512px-Pandas_logo.svg.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction-to-Pandas\" data-toc-modified-id=\"Introduction-to-Pandas-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction to Pandas</a></span><ul class=\"toc-item\"><li><span><a href=\"#Installation-and-Import\" data-toc-modified-id=\"Installation-and-Import-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Installation and Import</a></span></li></ul></li><li><span><a href=\"#Working-with-Series\" data-toc-modified-id=\"Working-with-Series-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Working with Series</a></span><ul class=\"toc-item\"><li><span><a href=\"#Creating-Series-from-a-List-or-Array\" data-toc-modified-id=\"Creating-Series-from-a-List-or-Array-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Creating Series from a List or Array</a></span></li><li><span><a href=\"#Creating-Series-from-a-Dictionary\" data-toc-modified-id=\"Creating-Series-from-a-Dictionary-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Creating Series from a Dictionary</a></span></li></ul></li><li><span><a href=\"#Working-with-DataFrame\" data-toc-modified-id=\"Working-with-DataFrame-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Working with DataFrame</a></span><ul class=\"toc-item\"><li><span><a href=\"#Creating-a-DataFrame\" data-toc-modified-id=\"Creating-a-DataFrame-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Creating a DataFrame</a></span><ul class=\"toc-item\"><li><span><a href=\"#From-a-Dictionary\" data-toc-modified-id=\"From-a-Dictionary-3.1.1\"><span class=\"toc-item-num\">3.1.1&nbsp;&nbsp;</span>From a Dictionary</a></span></li><li><span><a href=\"#From-a-Series\" data-toc-modified-id=\"From-a-Series-3.1.2\"><span class=\"toc-item-num\">3.1.2&nbsp;&nbsp;</span>From a Series</a></span></li><li><span><a href=\"#From-a-Dictionary-of-Series\" data-toc-modified-id=\"From-a-Dictionary-of-Series-3.1.3\"><span class=\"toc-item-num\">3.1.3&nbsp;&nbsp;</span>From a Dictionary of Series</a></span></li><li><span><a href=\"#From-a-two-dimensional-NumPy-Array\" data-toc-modified-id=\"From-a-two-dimensional-NumPy-Array-3.1.4\"><span class=\"toc-item-num\">3.1.4&nbsp;&nbsp;</span>From a two-dimensional NumPy Array</a></span></li></ul></li><li><span><a href=\"#Importing-Data-into-a-DataFrame\" data-toc-modified-id=\"Importing-Data-into-a-DataFrame-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Importing Data into a DataFrame</a></span><ul class=\"toc-item\"><li><span><a href=\"#Read-from-a-csv\" data-toc-modified-id=\"Read-from-a-csv-3.2.1\"><span class=\"toc-item-num\">3.2.1&nbsp;&nbsp;</span>Read from a csv</a></span></li><li><span><a href=\"#Read-from-an-excel-file\" data-toc-modified-id=\"Read-from-an-excel-file-3.2.2\"><span class=\"toc-item-num\">3.2.2&nbsp;&nbsp;</span>Read from an excel file</a></span></li><li><span><a href=\"#Read-from-a-json\" data-toc-modified-id=\"Read-from-a-json-3.2.3\"><span class=\"toc-item-num\">3.2.3&nbsp;&nbsp;</span>Read from a json</a></span></li><li><span><a href=\"#Read-from-an-API\" data-toc-modified-id=\"Read-from-an-API-3.2.4\"><span class=\"toc-item-num\">3.2.4&nbsp;&nbsp;</span>Read from an API</a></span></li></ul></li><li><span><a href=\"#Navigating-a-DataFrame\" data-toc-modified-id=\"Navigating-a-DataFrame-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Navigating a DataFrame</a></span><ul class=\"toc-item\"><li><span><a href=\"#Selecting-Columns\" data-toc-modified-id=\"Selecting-Columns-3.3.1\"><span class=\"toc-item-num\">3.3.1&nbsp;&nbsp;</span>Selecting Columns</a></span></li><li><span><a href=\"#Selection-using-loc-and-iloc\" data-toc-modified-id=\"Selection-using-loc-and-iloc-3.3.2\"><span class=\"toc-item-num\">3.3.2&nbsp;&nbsp;</span>Selection using <code>loc</code> and <code>iloc</code></a></span></li><li><span><a href=\"#Conditional-Selection\" data-toc-modified-id=\"Conditional-Selection-3.3.3\"><span class=\"toc-item-num\">3.3.3&nbsp;&nbsp;</span>Conditional Selection</a></span></li><li><span><a href=\"#Selection-using-query()\" data-toc-modified-id=\"Selection-using-query()-3.3.4\"><span class=\"toc-item-num\">3.3.4&nbsp;&nbsp;</span>Selection using <code>query()</code></a></span><ul class=\"toc-item\"><li><span><a href=\"#Subset-based-on-Numeric-Variable\" data-toc-modified-id=\"Subset-based-on-Numeric-Variable-3.3.4.1\"><span class=\"toc-item-num\">3.3.4.1&nbsp;&nbsp;</span>Subset based on Numeric Variable</a></span></li><li><span><a href=\"#Subset-based-on-Categorical-Variable\" data-toc-modified-id=\"Subset-based-on-Categorical-Variable-3.3.4.2\"><span class=\"toc-item-num\">3.3.4.2&nbsp;&nbsp;</span>Subset based on Categorical Variable</a></span></li><li><span><a href=\"#Subset-based-on-Multiple-Conditions\" data-toc-modified-id=\"Subset-based-on-Multiple-Conditions-3.3.4.3\"><span class=\"toc-item-num\">3.3.4.3&nbsp;&nbsp;</span>Subset based on Multiple Conditions</a></span></li><li><span><a href=\"#Escape-special-characters\" data-toc-modified-id=\"Escape-special-characters-3.3.4.4\"><span class=\"toc-item-num\">3.3.4.4&nbsp;&nbsp;</span>Escape special characters</a></span></li></ul></li></ul></li><li><span><a href=\"#Data-Types-in-a-DataFrame\" data-toc-modified-id=\"Data-Types-in-a-DataFrame-3.4\"><span class=\"toc-item-num\">3.4&nbsp;&nbsp;</span>Data Types in a DataFrame</a></span><ul class=\"toc-item\"><li><span><a href=\"#Checking-Data-Types\" data-toc-modified-id=\"Checking-Data-Types-3.4.1\"><span class=\"toc-item-num\">3.4.1&nbsp;&nbsp;</span>Checking Data Types</a></span></li><li><span><a href=\"#Changing-Data-Types\" data-toc-modified-id=\"Changing-Data-Types-3.4.2\"><span class=\"toc-item-num\">3.4.2&nbsp;&nbsp;</span>Changing Data Types</a></span><ul class=\"toc-item\"><li><span><a href=\"#Using-astype()-function\" data-toc-modified-id=\"Using-astype()-function-3.4.2.1\"><span class=\"toc-item-num\">3.4.2.1&nbsp;&nbsp;</span>Using <code>astype()</code> function</a></span></li><li><span><a href=\"#Custom-Functions\" data-toc-modified-id=\"Custom-Functions-3.4.2.2\"><span class=\"toc-item-num\">3.4.2.2&nbsp;&nbsp;</span>Custom Functions</a></span></li><li><span><a href=\"#Pandas-built-in-functions\" data-toc-modified-id=\"Pandas-built-in-functions-3.4.2.3\"><span class=\"toc-item-num\">3.4.2.3&nbsp;&nbsp;</span>Pandas built-in functions</a></span></li></ul></li></ul></li><li><span><a href=\"#Operating-on-DataFrames\" data-toc-modified-id=\"Operating-on-DataFrames-3.5\"><span class=\"toc-item-num\">3.5&nbsp;&nbsp;</span>Operating on DataFrames</a></span><ul class=\"toc-item\"><li><span><a href=\"#Arithmetic-Operations\" data-toc-modified-id=\"Arithmetic-Operations-3.5.1\"><span class=\"toc-item-num\">3.5.1&nbsp;&nbsp;</span>Arithmetic Operations</a></span><ul class=\"toc-item\"><li><span><a href=\"#Unary-Operations\" data-toc-modified-id=\"Unary-Operations-3.5.1.1\"><span class=\"toc-item-num\">3.5.1.1&nbsp;&nbsp;</span>Unary Operations</a></span></li><li><span><a href=\"#Binary-Operations\" data-toc-modified-id=\"Binary-Operations-3.5.1.2\"><span class=\"toc-item-num\">3.5.1.2&nbsp;&nbsp;</span>Binary Operations</a></span></li><li><span><a href=\"#Operations-between-DataFrame-and-Series\" data-toc-modified-id=\"Operations-between-DataFrame-and-Series-3.5.1.3\"><span class=\"toc-item-num\">3.5.1.3&nbsp;&nbsp;</span>Operations between DataFrame and Series</a></span></li></ul></li><li><span><a href=\"#Reindexing-Data\" data-toc-modified-id=\"Reindexing-Data-3.5.2\"><span class=\"toc-item-num\">3.5.2&nbsp;&nbsp;</span>Reindexing Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Reset-Index\" data-toc-modified-id=\"Reset-Index-3.5.2.1\"><span class=\"toc-item-num\">3.5.2.1&nbsp;&nbsp;</span>Reset Index</a></span></li></ul></li><li><span><a href=\"#Dropping-Data\" data-toc-modified-id=\"Dropping-Data-3.5.3\"><span class=\"toc-item-num\">3.5.3&nbsp;&nbsp;</span>Dropping Data</a></span></li><li><span><a href=\"#Adding-Data\" data-toc-modified-id=\"Adding-Data-3.5.4\"><span class=\"toc-item-num\">3.5.4&nbsp;&nbsp;</span>Adding Data</a></span></li><li><span><a href=\"#Renaming-Columns\" data-toc-modified-id=\"Renaming-Columns-3.5.5\"><span class=\"toc-item-num\">3.5.5&nbsp;&nbsp;</span>Renaming Columns</a></span></li></ul></li><li><span><a href=\"#Working-with-Missing-Data\" data-toc-modified-id=\"Working-with-Missing-Data-3.6\"><span class=\"toc-item-num\">3.6&nbsp;&nbsp;</span>Working with Missing Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Missing-Data-in-Pandas\" data-toc-modified-id=\"Missing-Data-in-Pandas-3.6.1\"><span class=\"toc-item-num\">3.6.1&nbsp;&nbsp;</span>Missing Data in Pandas</a></span><ul class=\"toc-item\"><li><span><a href=\"#None-object\" data-toc-modified-id=\"None-object-3.6.1.1\"><span class=\"toc-item-num\">3.6.1.1&nbsp;&nbsp;</span><code>None</code> object</a></span></li><li><span><a href=\"#NaN:-Not-a-Number\" data-toc-modified-id=\"NaN:-Not-a-Number-3.6.1.2\"><span class=\"toc-item-num\">3.6.1.2&nbsp;&nbsp;</span><code>NaN</code>: Not a Number</a></span></li><li><span><a href=\"#NaN-and-None\" data-toc-modified-id=\"NaN-and-None-3.6.1.3\"><span class=\"toc-item-num\">3.6.1.3&nbsp;&nbsp;</span><code>NaN</code> and <code>None</code></a></span></li></ul></li><li><span><a href=\"#Detect-Missing-Data\" data-toc-modified-id=\"Detect-Missing-Data-3.6.2\"><span class=\"toc-item-num\">3.6.2&nbsp;&nbsp;</span>Detect Missing Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Missing-values-in-each-column\" data-toc-modified-id=\"Missing-values-in-each-column-3.6.2.1\"><span class=\"toc-item-num\">3.6.2.1&nbsp;&nbsp;</span>Missing values in each column</a></span></li><li><span><a href=\"#Missing-values-in-DataFrame\" data-toc-modified-id=\"Missing-values-in-DataFrame-3.6.2.2\"><span class=\"toc-item-num\">3.6.2.2&nbsp;&nbsp;</span>Missing values in DataFrame</a></span></li><li><span><a href=\"#Using-notnull()\" data-toc-modified-id=\"Using-notnull()-3.6.2.3\"><span class=\"toc-item-num\">3.6.2.3&nbsp;&nbsp;</span>Using <code>notnull()</code></a></span></li></ul></li><li><span><a href=\"#Visualize-Missing-Data\" data-toc-modified-id=\"Visualize-Missing-Data-3.6.3\"><span class=\"toc-item-num\">3.6.3&nbsp;&nbsp;</span>Visualize Missing Data</a></span></li><li><span><a href=\"#Filter-based-on-Missing-Data\" data-toc-modified-id=\"Filter-based-on-Missing-Data-3.6.4\"><span class=\"toc-item-num\">3.6.4&nbsp;&nbsp;</span>Filter based on Missing Data</a></span></li><li><span><a href=\"#Drop-Missing-Data\" data-toc-modified-id=\"Drop-Missing-Data-3.6.5\"><span class=\"toc-item-num\">3.6.5&nbsp;&nbsp;</span>Drop Missing Data</a></span></li><li><span><a href=\"#Impute-Missing-Data\" data-toc-modified-id=\"Impute-Missing-Data-3.6.6\"><span class=\"toc-item-num\">3.6.6&nbsp;&nbsp;</span>Impute Missing Data</a></span><ul class=\"toc-item\"><li><span><a href=\"#Fill-with-a-Constant-Value\" data-toc-modified-id=\"Fill-with-a-Constant-Value-3.6.6.1\"><span class=\"toc-item-num\">3.6.6.1&nbsp;&nbsp;</span>Fill with a Constant Value</a></span></li><li><span><a href=\"#Forawrd-and-Backward-Fill\" data-toc-modified-id=\"Forawrd-and-Backward-Fill-3.6.6.2\"><span class=\"toc-item-num\">3.6.6.2&nbsp;&nbsp;</span>Forawrd and Backward Fill</a></span></li><li><span><a href=\"#Fill-with-Arithmetic-or-Aggregate-Functions\" data-toc-modified-id=\"Fill-with-Arithmetic-or-Aggregate-Functions-3.6.6.3\"><span class=\"toc-item-num\">3.6.6.3&nbsp;&nbsp;</span>Fill with Arithmetic or Aggregate Functions</a></span></li><li><span><a href=\"#Fill-using-a-Dictionary-of-Values\" data-toc-modified-id=\"Fill-using-a-Dictionary-of-Values-3.6.6.4\"><span class=\"toc-item-num\">3.6.6.4&nbsp;&nbsp;</span>Fill using a Dictionary of Values</a></span></li><li><span><a href=\"#Using-interpolate()\" data-toc-modified-id=\"Using-interpolate()-3.6.6.5\"><span class=\"toc-item-num\">3.6.6.5&nbsp;&nbsp;</span>Using <code>interpolate()</code></a></span></li></ul></li></ul></li></ul></li><li><span><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Conclusion</a></span></li><li><span><a href=\"#References\" data-toc-modified-id=\"References-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>References</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous notebook, we dove into the details on [NumPy](https://numpy.org/), which provides efficient storage and ability to perform complex computations through its `ndarray` object for working with homogeneous numerical array data. Here, we will build on the knowledge by looking into the data structures provided by __[Pandas](https://pandas.pydata.org/)__.\n",
    "\n",
    "__[Pandas](https://pandas.pydata.org/)__ provides high-level data structures and functions designed to make working with structured or tabular data fast, easy, and expressive. It is based on two main data structures:\n",
    "- `Series`: one-dimensional such as a list of items\n",
    "- `DataFrame`: two-dimensional, such as a table\n",
    "\n",
    "Both `Series` and `DataFrame` objects build on the NumPy array structure and form the core data model for Pandas in Python. While NumPy is best suited for working with homogeneous data, Pandas is designed for working with tabular or heterogeneous data. NumPy serves its purpose very well when the data is well organized and clean, however its limitations become clear when working with less structured data where we need more flexibility such as attaching labels to data, working with missing data, grouping, pivoting etc.\n",
    "\n",
    "Pandas blends the high-performance, array-computing ideas of NumPy with the flexible data manipulation capabilities of relational databases (such as SQL). Pandas provides sophisticated indexing functionality to reshape, slice and dice, perform aggregations, and select subsets of data. It also provides capabilities for easily handling missing data, adding/deleting columns, imputing missing data, and creating plots on the go. Pandas is a must-have tool for data wrangling and manipulation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installation and Import"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A typical installation of Python API comes with Pandas. You can import it as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also check the version of Pandas that is installed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.0.5'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Pandas `Series` is a one-dimensional array of indexed data. A `Series` object contains a sequence of values and an associated array of data labels, called `index`. While Numpy Array has an implicitly defined integer index that can be used to access the values, the index for a Pandas `Series` can also be explicitly defined. Let's take a look at some examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Series from a List or Array\n",
    "\n",
    "A `Series` can be created from a list or array as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    2\n",
       "2    3\n",
       "3    6\n",
       "4    7\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create Series\n",
    "data1 = [1,2,3,6,7]\n",
    "s1 = pd.Series(data1)\n",
    "s1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The array representation and index object of the `Series` can be accessed via its `values` and `index` attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Values in series s1: [1 2 3 6 7]\n",
      "Index of series s1: RangeIndex(start=0, stop=5, step=1)\n"
     ]
    }
   ],
   "source": [
    "print('Values in series s1:', s1.values)\n",
    "print('Index of series s1:', s1.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like NumPy arrays, data in a `Series` can be accessed by the associated index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "6\n"
     ]
    }
   ],
   "source": [
    "# Access elements of a Series\n",
    "print(s1[0])\n",
    "print(s1[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For `Series` objects, the index need not be an integer and can be explicitly defined as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "d    1\n",
       "b    2\n",
       "a    3\n",
       "c    6\n",
       "e    7\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s2 = pd.Series([1,2,3,6,7], index=['d', 'b', 'a', 'c', 'e'])\n",
    "s2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['d', 'b', 'a', 'c', 'e'], dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the index\n",
    "s2.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Element associated with index a: 3\n",
      "Element associated with index e: 7\n"
     ]
    }
   ],
   "source": [
    "# Access elements using index label\n",
    "\n",
    "print('Element associated with index a:', s2['a'])\n",
    "print('Element associated with index e:', s2['e'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Element associated with index position 2: 3\n",
      "Element associated with last index position: 7\n"
     ]
    }
   ],
   "source": [
    "# Access elements using index position\n",
    "\n",
    "print('Element associated with index position 2:', s2[2])\n",
    "print('Element associated with last index position:', s2[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Series from a Dictionary\n",
    "\n",
    "Just like a dictionary maps keys to a set of values, a series can be thought of as a mapping of index values to data values. Let's create a series from a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA    35000\n",
       "TX    71000\n",
       "WA    16000\n",
       "OR     5000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {'CA': 35000, 'TX': 71000, 'WA': 16000, 'OR': 5000}\n",
    "s3 = pd.Series(data)\n",
    "s3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, index for `Series` will be created using the keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "35000"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Access elements using index label\n",
    "s3['CA']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "35000"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Access elements using index position\n",
    "s3[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A [`DataFrame`](https://pandas.pydata.org/pandas-docs/stable/user_guide/dsintro.html) represents a rectangular table of data and contains an ordered collection of columns. You can think of it as a spreadsheet or SQL table where each column has a column name for reference and each row can be accessed by using row numbers. Column names and row numbers are known as column and row index.\n",
    "\n",
    "DataFrame is a fundamental __Pandas__ data structure in which each column can be of a different value type (numeric, string, boolean, etc.). A data set can be first read into a DataFrame and then various operations (i.e. indexing, grouping, aggregation etc.) can be easily applied to it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a DataFrame\n",
    "There are many ways to construct a DataFrame. Let's look at some examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### From a Dictionary\n",
    "`DataFrame` can be constructed from a dictionary of equal sized lists or NumPy arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>state</th>\n",
       "      <th>year</th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CA</td>\n",
       "      <td>2015</td>\n",
       "      <td>38.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>WA</td>\n",
       "      <td>2015</td>\n",
       "      <td>7.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CA</td>\n",
       "      <td>2016</td>\n",
       "      <td>39.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>WA</td>\n",
       "      <td>2016</td>\n",
       "      <td>7.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CA</td>\n",
       "      <td>2017</td>\n",
       "      <td>39.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>WA</td>\n",
       "      <td>2017</td>\n",
       "      <td>7.50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  state  year  population\n",
       "0    CA  2015       38.50\n",
       "1    WA  2015        7.00\n",
       "2    CA  2016       39.00\n",
       "3    WA  2016        7.25\n",
       "4    CA  2017       39.50\n",
       "5    WA  2017        7.50"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {'state':['CA','WA','CA','WA','CA','WA'],\n",
    "        'year':[2015,2015,2016,2016,2017,2017],\n",
    "        'population':[38.5,7.0,39.0,7.25,39.5,7.5]}\n",
    "\n",
    "df1 = pd.DataFrame(data)\n",
    "df1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### From a Series\n",
    "`DataFrame` is a collection of `Series` objects. Let's look at constructing a `DataFrame` from a single `Series` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Population series\n",
    "pop_data = {'CA': 39.5,'TX': 29,'NY': 8.39,'FL': 21.48,\n",
    "            'IL': 12.67}\n",
    "population = pd.Series(pop_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>8.39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population\n",
       "CA       39.50\n",
       "TX       29.00\n",
       "NY        8.39\n",
       "FL       21.48\n",
       "IL       12.67"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create DataFrame from series\n",
    "pop_df = pd.DataFrame(population, columns=['population'])\n",
    "pop_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### From a Dictionary of Series\n",
    "`DataFrame` can also be constructed from a dictionary of `Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA    155779.22\n",
       "TX    261231.71\n",
       "NY     47126.40\n",
       "FL     53624.76\n",
       "IL     55518.93\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create Area series\n",
    "area_data = {'CA': 155779.22, 'TX': 261231.71, 'NY': 47126.40,\n",
    "             'FL': 53624.76, 'IL': 55518.93}\n",
    "area = pd.Series(area_data)\n",
    "area"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.50</td>\n",
       "      <td>155779.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.00</td>\n",
       "      <td>261231.71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>8.39</td>\n",
       "      <td>47126.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area\n",
       "CA       39.50  155779.22\n",
       "TX       29.00  261231.71\n",
       "NY        8.39   47126.40\n",
       "FL       21.48   53624.76\n",
       "IL       12.67   55518.93"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create DataFrame from dictionary of Series\n",
    "df2 = pd.DataFrame({'population':population,\n",
    "                   'land area':area})\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### From a two-dimensional NumPy Array\n",
    "`DataFrame` can be constructed from a two-dimensional NumPy array by specifying the column names. An integer `Index` is used if not specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Col_A</th>\n",
       "      <th>Col_B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0.028278</td>\n",
       "      <td>0.687096</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>0.999634</td>\n",
       "      <td>0.443590</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>0.750714</td>\n",
       "      <td>0.785287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>0.611652</td>\n",
       "      <td>0.651515</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Col_A     Col_B\n",
       "a  0.028278  0.687096\n",
       "b  0.999634  0.443590\n",
       "c  0.750714  0.785287\n",
       "d  0.611652  0.651515"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(np.random.rand(4, 2),\n",
    "             columns=['Col_A', 'Col_B'],\n",
    "             index=['a', 'b', 'c', 'd'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Data into a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reading data into a `DataFrame` is one of the most common task in any data science problem. __Pandas__ provides the ability to read data from various formats such as CSV, JSON, Excel, APIs, etc. directly into a `DataFrame` object. Let's look at how to read data from some common formats into a `DataFrame`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read from a csv\n",
    "`read_csv()` function can be used to read csv (comma-separated value) files. The function includes a number of different parameters and you can read more about them in the pandas documentation [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).\n",
    "\n",
    "Let's import the 'health.csv' file we used earlier in this guide series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Number of Beds</th>\n",
       "      <th>Name</th>\n",
       "      <th>Address</th>\n",
       "      <th>City</th>\n",
       "      <th>State</th>\n",
       "      <th>Zip Code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>156</td>\n",
       "      <td>Facility 1</td>\n",
       "      <td>2468 SOUTH ST ANDREWS PLACE</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59</td>\n",
       "      <td>Facility 2</td>\n",
       "      <td>2300 W. WASHINGTON BLVD.</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25</td>\n",
       "      <td>Facility 3</td>\n",
       "      <td>4060 E. WHITTIER BLVD.</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49</td>\n",
       "      <td>Facility 4</td>\n",
       "      <td>6070 W. PICO BOULEVARD</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55</td>\n",
       "      <td>Facility 5</td>\n",
       "      <td>1480 S. LA CIENEGA BL</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90035</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Number of Beds        Name                      Address         City State  \\\n",
       "0             156  Facility 1  2468 SOUTH ST ANDREWS PLACE  LOS ANGELES    CA   \n",
       "1              59  Facility 2     2300 W. WASHINGTON BLVD.  LOS ANGELES    CA   \n",
       "2              25  Facility 3       4060 E. WHITTIER BLVD.  LOS ANGELES    CA   \n",
       "3              49  Facility 4       6070 W. PICO BOULEVARD  LOS ANGELES    CA   \n",
       "4              55  Facility 5        1480 S. LA CIENEGA BL  LOS ANGELES    CA   \n",
       "\n",
       "   Zip Code  \n",
       "0     90018  \n",
       "1     90018  \n",
       "2     90023  \n",
       "3     90035  \n",
       "4     90035  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the data from csv\n",
    "df_csv = pd.read_csv('../data/health.csv')\n",
    "df_csv.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Confirm type\n",
    "type(df_csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read from an excel file\n",
    "Tabular data is often stored using Microsoft Excel 2003 (and higher) and can be read using `read_excel()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Number of Beds</th>\n",
       "      <th>Name</th>\n",
       "      <th>Address</th>\n",
       "      <th>City</th>\n",
       "      <th>State</th>\n",
       "      <th>Zip Code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>156</td>\n",
       "      <td>Facility 1</td>\n",
       "      <td>2468 SOUTH ST ANDREWS PLACE</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59</td>\n",
       "      <td>Facility 2</td>\n",
       "      <td>2300 W. WASHINGTON BLVD.</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25</td>\n",
       "      <td>Facility 3</td>\n",
       "      <td>4060 E. WHITTIER BLVD.</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49</td>\n",
       "      <td>Facility 4</td>\n",
       "      <td>6070 W. PICO BOULEVARD</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55</td>\n",
       "      <td>Facility 5</td>\n",
       "      <td>1480 S. LA CIENEGA BL</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90035</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Number of Beds        Name                      Address         City State  \\\n",
       "0             156  Facility 1  2468 SOUTH ST ANDREWS PLACE  LOS ANGELES    CA   \n",
       "1              59  Facility 2     2300 W. WASHINGTON BLVD.  LOS ANGELES    CA   \n",
       "2              25  Facility 3       4060 E. WHITTIER BLVD.  LOS ANGELES    CA   \n",
       "3              49  Facility 4       6070 W. PICO BOULEVARD  LOS ANGELES    CA   \n",
       "4              55  Facility 5        1480 S. LA CIENEGA BL  LOS ANGELES    CA   \n",
       "\n",
       "   Zip Code  \n",
       "0     90018  \n",
       "1     90018  \n",
       "2     90023  \n",
       "3     90035  \n",
       "4     90035  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the data from excel\n",
    "df_excel = pd.read_excel('../data/health.xlsx')\n",
    "df_excel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Confirm type\n",
    "type(df_excel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read from a json\n",
    "`read_json()` can be used to read JSON (JavaScript Object Notation) files. JSON is mostly used to store unstructured data with key/value pairs. The function accepts a valid JSON string, path object or file-like object and does not consume a dictionary (key/value pair) directly. Let's read 'health.json' into a `DataFrame`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Number of Beds</th>\n",
       "      <th>Name</th>\n",
       "      <th>Address</th>\n",
       "      <th>City</th>\n",
       "      <th>State</th>\n",
       "      <th>Zip Code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>156</td>\n",
       "      <td>Facility 1</td>\n",
       "      <td>2468 SOUTH ST ANDREWS PLACE</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>59</td>\n",
       "      <td>Facility 2</td>\n",
       "      <td>2300 W. WASHINGTON BLVD.</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90018</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>25</td>\n",
       "      <td>Facility 3</td>\n",
       "      <td>4060 E. WHITTIER BLVD.</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49</td>\n",
       "      <td>Facility 4</td>\n",
       "      <td>6070 W. PICO BOULEVARD</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55</td>\n",
       "      <td>Facility 5</td>\n",
       "      <td>1480 S. LA CIENEGA BL</td>\n",
       "      <td>LOS ANGELES</td>\n",
       "      <td>CA</td>\n",
       "      <td>90035</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Number of Beds        Name                      Address         City State  \\\n",
       "0             156  Facility 1  2468 SOUTH ST ANDREWS PLACE  LOS ANGELES    CA   \n",
       "1              59  Facility 2     2300 W. WASHINGTON BLVD.  LOS ANGELES    CA   \n",
       "2              25  Facility 3       4060 E. WHITTIER BLVD.  LOS ANGELES    CA   \n",
       "3              49  Facility 4       6070 W. PICO BOULEVARD  LOS ANGELES    CA   \n",
       "4              55  Facility 5        1480 S. LA CIENEGA BL  LOS ANGELES    CA   \n",
       "\n",
       "   Zip Code  \n",
       "0     90018  \n",
       "1     90018  \n",
       "2     90023  \n",
       "3     90035  \n",
       "4     90035  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the data from json\n",
    "df_json = pd.read_json('../data/health.json')\n",
    "df_json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Confirm type\n",
    "type(df_json)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read from an API\n",
    "Many websites provide data through public APIs in json or other formats. One easy way to access these APIs from Python is using `requests` package. Let's find issues for pandas on GitHub using the add-on requests library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import library\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make request and store response\n",
    "url = 'https://api.github.com/repos/pandas-dev/pandas/issues'\n",
    "resp = requests.get(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The response object’s `json` method will return a dictionary containing JSON. `.keys()` method can be used to explore the structure of the returned JSON object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['url', 'repository_url', 'labels_url', 'comments_url', 'events_url', 'html_url', 'id', 'node_id', 'number', 'title', 'user', 'labels', 'state', 'locked', 'assignee', 'assignees', 'milestone', 'comments', 'created_at', 'updated_at', 'closed_at', 'author_association', 'active_lock_reason', 'pull_request', 'body', 'performed_via_github_app'])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get json\n",
    "json_data = resp.json()\n",
    "json_data[0].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since each element in `json_data` is a dictionary, you can create a DataFrame using specific columns from the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>number</th>\n",
       "      <th>title</th>\n",
       "      <th>labels</th>\n",
       "      <th>state</th>\n",
       "      <th>created_at</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>38070</td>\n",
       "      <td>BUG: Index.drop raising Error when Index has d...</td>\n",
       "      <td>[]</td>\n",
       "      <td>open</td>\n",
       "      <td>2020-11-25T20:11:11Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>38069</td>\n",
       "      <td>ENH: Rename multi-level columns or indices usi...</td>\n",
       "      <td>[{'id': 76812, 'node_id': 'MDU6TGFiZWw3NjgxMg=...</td>\n",
       "      <td>open</td>\n",
       "      <td>2020-11-25T19:57:28Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38068</td>\n",
       "      <td>ENH: NDArrayBackedExtensionArray.__array_funct...</td>\n",
       "      <td>[]</td>\n",
       "      <td>open</td>\n",
       "      <td>2020-11-25T18:32:41Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>38067</td>\n",
       "      <td>DOC: Include missing holiday observance rules</td>\n",
       "      <td>[{'id': 134699, 'node_id': 'MDU6TGFiZWwxMzQ2OT...</td>\n",
       "      <td>open</td>\n",
       "      <td>2020-11-25T17:25:35Z</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>38066</td>\n",
       "      <td>RLS: 1.1.5</td>\n",
       "      <td>[{'id': 131473665, 'node_id': 'MDU6TGFiZWwxMzE...</td>\n",
       "      <td>open</td>\n",
       "      <td>2020-11-25T15:51:33Z</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   number                                              title  \\\n",
       "0   38070  BUG: Index.drop raising Error when Index has d...   \n",
       "1   38069  ENH: Rename multi-level columns or indices usi...   \n",
       "2   38068  ENH: NDArrayBackedExtensionArray.__array_funct...   \n",
       "3   38067      DOC: Include missing holiday observance rules   \n",
       "4   38066                                         RLS: 1.1.5   \n",
       "\n",
       "                                              labels state  \\\n",
       "0                                                 []  open   \n",
       "1  [{'id': 76812, 'node_id': 'MDU6TGFiZWw3NjgxMg=...  open   \n",
       "2                                                 []  open   \n",
       "3  [{'id': 134699, 'node_id': 'MDU6TGFiZWwxMzQ2OT...  open   \n",
       "4  [{'id': 131473665, 'node_id': 'MDU6TGFiZWwxMzE...  open   \n",
       "\n",
       "             created_at  \n",
       "0  2020-11-25T20:11:11Z  \n",
       "1  2020-11-25T19:57:28Z  \n",
       "2  2020-11-25T18:32:41Z  \n",
       "3  2020-11-25T17:25:35Z  \n",
       "4  2020-11-25T15:51:33Z  "
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create dataframe for specific cols\n",
    "issues = pd.DataFrame(json_data, columns=['number', 'title',\n",
    "                                   'labels', 'state', 'created_at'])\n",
    "issues.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a `DataFrame` with all the data directly from the response object, __\"json\"__ library can be used. Since `read_json()` accepts a valid JSON string, `json.dumps()` can be used to convert the object back to a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import library\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>repository_url</th>\n",
       "      <th>labels_url</th>\n",
       "      <th>comments_url</th>\n",
       "      <th>events_url</th>\n",
       "      <th>html_url</th>\n",
       "      <th>id</th>\n",
       "      <th>node_id</th>\n",
       "      <th>number</th>\n",
       "      <th>title</th>\n",
       "      <th>...</th>\n",
       "      <th>milestone</th>\n",
       "      <th>comments</th>\n",
       "      <th>created_at</th>\n",
       "      <th>updated_at</th>\n",
       "      <th>closed_at</th>\n",
       "      <th>author_association</th>\n",
       "      <th>active_lock_reason</th>\n",
       "      <th>pull_request</th>\n",
       "      <th>body</th>\n",
       "      <th>performed_via_github_app</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://github.com/pandas-dev/pandas/pull/38070</td>\n",
       "      <td>751090409</td>\n",
       "      <td>MDExOlB1bGxSZXF1ZXN0NTI3NjcwMDk4</td>\n",
       "      <td>38070</td>\n",
       "      <td>BUG: Index.drop raising Error when Index has d...</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>2020-11-25 20:11:11+00:00</td>\n",
       "      <td>2020-11-25 20:39:16+00:00</td>\n",
       "      <td>NaT</td>\n",
       "      <td>MEMBER</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'url': 'https://api.github.com/repos/pandas-d...</td>\n",
       "      <td>- [x] closes #38051\\r\\n- [x] closes #33494\\r\\n...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://github.com/pandas-dev/pandas/issues/38069</td>\n",
       "      <td>751083108</td>\n",
       "      <td>MDU6SXNzdWU3NTEwODMxMDg=</td>\n",
       "      <td>38069</td>\n",
       "      <td>ENH: Rename multi-level columns or indices usi...</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>2020-11-25 19:57:28+00:00</td>\n",
       "      <td>2020-11-25 19:57:28+00:00</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NONE</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>It's currently quite difficult to rename a sin...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://github.com/pandas-dev/pandas/pull/38068</td>\n",
       "      <td>751038480</td>\n",
       "      <td>MDExOlB1bGxSZXF1ZXN0NTI3NjI3Njc2</td>\n",
       "      <td>38068</td>\n",
       "      <td>ENH: NDArrayBackedExtensionArray.__array_funct...</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>1</td>\n",
       "      <td>2020-11-25 18:32:41+00:00</td>\n",
       "      <td>2020-11-25 18:49:53+00:00</td>\n",
       "      <td>NaT</td>\n",
       "      <td>MEMBER</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'url': 'https://api.github.com/repos/pandas-d...</td>\n",
       "      <td>motivated by getting np.delete and np.repeat w...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://github.com/pandas-dev/pandas/issues/38067</td>\n",
       "      <td>750998859</td>\n",
       "      <td>MDU6SXNzdWU3NTA5OTg4NTk=</td>\n",
       "      <td>38067</td>\n",
       "      <td>DOC: Include missing holiday observance rules</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>2020-11-25 17:25:35+00:00</td>\n",
       "      <td>2020-11-25 17:28:13+00:00</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NONE</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>#### Location of the documentation\\r\\n\\r\\n[Tim...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://api.github.com/repos/pandas-dev/pandas...</td>\n",
       "      <td>https://github.com/pandas-dev/pandas/issues/38066</td>\n",
       "      <td>750932114</td>\n",
       "      <td>MDU6SXNzdWU3NTA5MzIxMTQ=</td>\n",
       "      <td>38066</td>\n",
       "      <td>RLS: 1.1.5</td>\n",
       "      <td>...</td>\n",
       "      <td>{'url': 'https://api.github.com/repos/pandas-d...</td>\n",
       "      <td>3</td>\n",
       "      <td>2020-11-25 15:51:33+00:00</td>\n",
       "      <td>2020-11-25 16:11:36+00:00</td>\n",
       "      <td>NaT</td>\n",
       "      <td>MEMBER</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Tracking issue for the 1.1.5 release.\\r\\n\\r\\nh...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 url  \\\n",
       "0  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "1  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "2  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "3  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "4  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "\n",
       "                                   repository_url  \\\n",
       "0  https://api.github.com/repos/pandas-dev/pandas   \n",
       "1  https://api.github.com/repos/pandas-dev/pandas   \n",
       "2  https://api.github.com/repos/pandas-dev/pandas   \n",
       "3  https://api.github.com/repos/pandas-dev/pandas   \n",
       "4  https://api.github.com/repos/pandas-dev/pandas   \n",
       "\n",
       "                                          labels_url  \\\n",
       "0  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "1  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "2  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "3  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "4  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "\n",
       "                                        comments_url  \\\n",
       "0  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "1  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "2  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "3  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "4  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "\n",
       "                                          events_url  \\\n",
       "0  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "1  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "2  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "3  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "4  https://api.github.com/repos/pandas-dev/pandas...   \n",
       "\n",
       "                                            html_url         id  \\\n",
       "0    https://github.com/pandas-dev/pandas/pull/38070  751090409   \n",
       "1  https://github.com/pandas-dev/pandas/issues/38069  751083108   \n",
       "2    https://github.com/pandas-dev/pandas/pull/38068  751038480   \n",
       "3  https://github.com/pandas-dev/pandas/issues/38067  750998859   \n",
       "4  https://github.com/pandas-dev/pandas/issues/38066  750932114   \n",
       "\n",
       "                            node_id  number  \\\n",
       "0  MDExOlB1bGxSZXF1ZXN0NTI3NjcwMDk4   38070   \n",
       "1          MDU6SXNzdWU3NTEwODMxMDg=   38069   \n",
       "2  MDExOlB1bGxSZXF1ZXN0NTI3NjI3Njc2   38068   \n",
       "3          MDU6SXNzdWU3NTA5OTg4NTk=   38067   \n",
       "4          MDU6SXNzdWU3NTA5MzIxMTQ=   38066   \n",
       "\n",
       "                                               title  ...  \\\n",
       "0  BUG: Index.drop raising Error when Index has d...  ...   \n",
       "1  ENH: Rename multi-level columns or indices usi...  ...   \n",
       "2  ENH: NDArrayBackedExtensionArray.__array_funct...  ...   \n",
       "3      DOC: Include missing holiday observance rules  ...   \n",
       "4                                         RLS: 1.1.5  ...   \n",
       "\n",
       "                                           milestone comments  \\\n",
       "0                                               None        0   \n",
       "1                                               None        0   \n",
       "2                                               None        1   \n",
       "3                                               None        0   \n",
       "4  {'url': 'https://api.github.com/repos/pandas-d...        3   \n",
       "\n",
       "                 created_at                updated_at  closed_at  \\\n",
       "0 2020-11-25 20:11:11+00:00 2020-11-25 20:39:16+00:00        NaT   \n",
       "1 2020-11-25 19:57:28+00:00 2020-11-25 19:57:28+00:00        NaT   \n",
       "2 2020-11-25 18:32:41+00:00 2020-11-25 18:49:53+00:00        NaT   \n",
       "3 2020-11-25 17:25:35+00:00 2020-11-25 17:28:13+00:00        NaT   \n",
       "4 2020-11-25 15:51:33+00:00 2020-11-25 16:11:36+00:00        NaT   \n",
       "\n",
       "  author_association active_lock_reason  \\\n",
       "0             MEMBER                NaN   \n",
       "1               NONE                NaN   \n",
       "2             MEMBER                NaN   \n",
       "3               NONE                NaN   \n",
       "4             MEMBER                NaN   \n",
       "\n",
       "                                        pull_request  \\\n",
       "0  {'url': 'https://api.github.com/repos/pandas-d...   \n",
       "1                                                NaN   \n",
       "2  {'url': 'https://api.github.com/repos/pandas-d...   \n",
       "3                                                NaN   \n",
       "4                                                NaN   \n",
       "\n",
       "                                                body performed_via_github_app  \n",
       "0  - [x] closes #38051\\r\\n- [x] closes #33494\\r\\n...                      NaN  \n",
       "1  It's currently quite difficult to rename a sin...                      NaN  \n",
       "2  motivated by getting np.delete and np.repeat w...                      NaN  \n",
       "3  #### Location of the documentation\\r\\n\\r\\n[Tim...                      NaN  \n",
       "4  Tracking issue for the 1.1.5 release.\\r\\n\\r\\nh...                      NaN  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read all data from response object's json method\n",
    "all_issues = pd.read_json(json.dumps(json_data))\n",
    "all_issues.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Navigating a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Elements or subsets of a `DataFrame` can be accessed in multiple ways. We can use `[]` or use slice notation, marked by the colon (`:`) character to access subsets of data. Indexing and slicing to select subsets of a DataFrame can be performed:\n",
    "- By location, like lists\n",
    "- By label, like dictionaries\n",
    "- By boolean masks like NumPy arrays\n",
    "\n",
    "Special indexing operators such as `loc` and `iloc` also enable selection of a subset of the rows and columns from a DataFrame. Let's look at some examples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting Columns\n",
    "Columns of the DataFrame are essentially `Series` objects that can be accessed via dictionary-style indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA    39.50\n",
       "TX    29.00\n",
       "NY     8.39\n",
       "FL    21.48\n",
       "IL    12.67\n",
       "Name: population, dtype: float64"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['population']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Individual columns can also be selected using attribute-style access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA    39.50\n",
       "TX    29.00\n",
       "NY     8.39\n",
       "FL    21.48\n",
       "IL    12.67\n",
       "Name: population, dtype: float64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.population"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selection using `loc` and `iloc`\n",
    "\n",
    "Special indexing operators such as `loc` and `iloc` can be used to select a subset of the rows and columns from a DataFrame. \n",
    "\n",
    "- `.loc` for label-based indexing can be used to index the data in an array-like style by specifying index and column names\n",
    "- `.iloc` for positional indexing can be used to index the underlying array as if it is a simple NumPy array.\n",
    "\n",
    "Let's add another column to `df2` DataFrame and then look at some examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.50</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.00</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>8.39</td>\n",
       "      <td>47126.40</td>\n",
       "      <td>178.031846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "      <td>400.561233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "      <td>228.210450</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density\n",
       "CA       39.50  155779.22  253.563986\n",
       "TX       29.00  261231.71  111.012557\n",
       "NY        8.39   47126.40  178.031846\n",
       "FL       21.48   53624.76  400.561233\n",
       "IL       12.67   55518.93  228.210450"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add column\n",
    "df2['density'] = df2['population']*1000000 / df2['land area']\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Selection using `.iloc`__\n",
    "\n",
    "`.iloc` is used for positional indexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.5</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.0</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density\n",
       "CA        39.5  155779.22  253.563986\n",
       "TX        29.0  261231.71  111.012557"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First two rows and all columns\n",
    "df2.iloc[:2, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area\n",
       "FL       21.48   53624.76\n",
       "IL       12.67   55518.93"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Last two rows and first two columns\n",
    "df2.iloc[-2:, :2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>land area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>47126.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>53624.76</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    land area\n",
       "NY   47126.40\n",
       "FL   53624.76"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Middle subset of rows and columns\n",
    "df2.iloc[2:4, 1:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Selection using `.loc`__\n",
    "\n",
    "`.loc` is used for label-based indexing by specifying index and column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.5</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.0</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density\n",
       "CA        39.5  155779.22  253.563986\n",
       "TX        29.0  261231.71  111.012557"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select rows until 'TX' and all columns\n",
    "df2.loc[:'TX', :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area\n",
       "FL       21.48   53624.76\n",
       "IL       12.67   55518.93"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select rows from 'FL' and columns until 'area'\n",
    "df2.loc['FL':, :'land area']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>47126.40</td>\n",
       "      <td>178.031846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>53624.76</td>\n",
       "      <td>400.561233</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    land area     density\n",
       "NY   47126.40  178.031846\n",
       "FL   53624.76  400.561233"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Middle subset of rows from NY to FL and columns from 'area' to 'density'\n",
    "df2.loc['NY':'FL', 'land area':'density']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conditional Selection\n",
    "\n",
    "_Boolean masks_ can be used to conditionally select specific subsets of the data. They select the elements of an `DataFrame` that satisfy some condition. Let's take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA     True\n",
       "TX    False\n",
       "NY    False\n",
       "FL     True\n",
       "IL     True\n",
       "Name: density, dtype: bool"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['density'] > 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.50</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "      <td>400.561233</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "      <td>228.210450</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density\n",
       "CA       39.50  155779.22  253.563986\n",
       "FL       21.48   53624.76  400.561233\n",
       "IL       12.67   55518.93  228.210450"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select data where density > 200\n",
    "df2[df2['density'] > 200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.5</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.0</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density\n",
       "CA        39.5  155779.22  253.563986\n",
       "TX        29.0  261231.71  111.012557"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Select data for only those state where area > 50000 and return first 2 rows\n",
    "df2[df2['land area'] > 50000].iloc[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selection using `query()`\n",
    "\n",
    "`query()` enables you to “query” a DataFrame and retrieve subsets based on logical conditions. It acts similar to the ‘where’ clause in SQL making it much easier to read and understand. `query()` uses string expressions to efficiently compute operations on a DataFrame and offers a more efficient computation compared to the masking expression. It also provides efficient memory use than pure python operations.\n",
    "\n",
    "Let's add some data to `df2` and take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['drought'] = ['Yes','Yes','No','Yes','No']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "      <th>drought</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.50</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.00</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>8.39</td>\n",
       "      <td>47126.40</td>\n",
       "      <td>178.031846</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "      <td>400.561233</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "      <td>228.210450</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density drought\n",
       "CA       39.50  155779.22  253.563986     Yes\n",
       "TX       29.00  261231.71  111.012557     Yes\n",
       "NY        8.39   47126.40  178.031846      No\n",
       "FL       21.48   53624.76  400.561233     Yes\n",
       "IL       12.67   55518.93  228.210450      No"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Subset based on Numeric Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "      <th>drought</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.50</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.00</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "      <td>400.561233</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density drought\n",
       "CA       39.50  155779.22  253.563986     Yes\n",
       "TX       29.00  261231.71  111.012557     Yes\n",
       "FL       21.48   53624.76  400.561233     Yes"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('population > 20')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Subset based on Categorical Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "      <th>drought</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>8.39</td>\n",
       "      <td>47126.40</td>\n",
       "      <td>178.031846</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "      <td>228.210450</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density drought\n",
       "NY        8.39   47126.40  178.031846      No\n",
       "IL       12.67   55518.93  228.210450      No"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('drought == \"No\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Subset based on Multiple Conditions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Condition 1__: population > 20 and density < 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "      <th>drought</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.0</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density drought\n",
       "TX        29.0  261231.71  111.012557     Yes"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# AND operator\n",
    "df2.query('(population > 20) and (density < 200)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Condition 2__: population < 25 or drought == \"No\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "      <th>drought</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>8.39</td>\n",
       "      <td>47126.40</td>\n",
       "      <td>178.031846</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "      <td>400.561233</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "      <td>228.210450</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density drought\n",
       "NY        8.39   47126.40  178.031846      No\n",
       "FL       21.48   53624.76  400.561233     Yes\n",
       "IL       12.67   55518.93  228.210450      No"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# OR operator\n",
    "df2.query('(population < 25) or (drought == \"No\")')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Condition 3__: population < 20 and index in [\"NY\", \"IL\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "      <th>drought</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>8.39</td>\n",
       "      <td>47126.40</td>\n",
       "      <td>178.031846</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "      <td>228.210450</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density drought\n",
       "NY        8.39   47126.40  178.031846      No\n",
       "IL       12.67   55518.93  228.210450      No"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# IN operator\n",
    "df2.query('(population < 20) and (index in [\"NY\", \"IL\"])')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Condition 4__: Like operation\n",
    "\n",
    "Although \"like\" is not supported, it can be simulated using string operations and specifying `engine='python'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "      <th>drought</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.50</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.00</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "      <td>400.561233</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density drought\n",
       "CA       39.50  155779.22  253.563986     Yes\n",
       "TX       29.00  261231.71  111.012557     Yes\n",
       "FL       21.48   53624.76  400.561233     Yes"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('drought.str.contains(\"e\")', engine='python')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Escape special characters\n",
    "\n",
    "`land area` column has a whitespace which may cause issues when a query gets executed. Wrap column names in backticks to escape special characters such as whitespace etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "      <th>drought</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.5</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.0</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "      <td>Yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density drought\n",
       "CA        39.5  155779.22  253.563986     Yes\n",
       "TX        29.0  261231.71  111.012557     Yes"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('`land area` > 60000')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Types in a DataFrame\n",
    "\n",
    "The way information is stored in a `DataFrame` affects what we can do with it and the outputs of calculations on it. When doing data analysis, it is important to make sure you are using the correct data types; otherwise you may get unexpected results or errors. Common data types available in __Pandas__ are `object`, `int64`, `float64`, `datetime64` and `bool`. \n",
    "\n",
    "An `object` data type can contain multiple different types such as integers, floats and strings. These different data types when included in a single column are collectively labeled as an `object`.\n",
    "\n",
    "Let's expand the `df2` DataFrame to add `month, year, date, gdp, rainfall and drought` columns and explore various data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>population</th>\n",
       "      <th>land area</th>\n",
       "      <th>density</th>\n",
       "      <th>drought</th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>gdp</th>\n",
       "      <th>rainfall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>39.50</td>\n",
       "      <td>155779.22</td>\n",
       "      <td>253.563986</td>\n",
       "      <td>Yes</td>\n",
       "      <td>2019</td>\n",
       "      <td>2</td>\n",
       "      <td>27</td>\n",
       "      <td>3183</td>\n",
       "      <td>22.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TX</th>\n",
       "      <td>29.00</td>\n",
       "      <td>261231.71</td>\n",
       "      <td>111.012557</td>\n",
       "      <td>Yes</td>\n",
       "      <td>2019</td>\n",
       "      <td>5</td>\n",
       "      <td>12</td>\n",
       "      <td>1918</td>\n",
       "      <td>35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NY</th>\n",
       "      <td>8.39</td>\n",
       "      <td>47126.40</td>\n",
       "      <td>178.031846</td>\n",
       "      <td>No</td>\n",
       "      <td>2020</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>1751</td>\n",
       "      <td>42.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FL</th>\n",
       "      <td>21.48</td>\n",
       "      <td>53624.76</td>\n",
       "      <td>400.561233</td>\n",
       "      <td>Yes</td>\n",
       "      <td>2019</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>1111</td>\n",
       "      <td>54.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IL</th>\n",
       "      <td>12.67</td>\n",
       "      <td>55518.93</td>\n",
       "      <td>228.210450</td>\n",
       "      <td>No</td>\n",
       "      <td>2020</td>\n",
       "      <td>11</td>\n",
       "      <td>17</td>\n",
       "      <td>908</td>\n",
       "      <td>No Value</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    population  land area     density drought  year  month  day   gdp  \\\n",
       "CA       39.50  155779.22  253.563986     Yes  2019      2   27  3183   \n",
       "TX       29.00  261231.71  111.012557     Yes  2019      5   12  1918   \n",
       "NY        8.39   47126.40  178.031846      No  2020      7   10  1751   \n",
       "FL       21.48   53624.76  400.561233     Yes  2019      9    7  1111   \n",
       "IL       12.67   55518.93  228.210450      No  2020     11   17   908   \n",
       "\n",
       "    rainfall  \n",
       "CA      22.9  \n",
       "TX        35  \n",
       "NY     42.87  \n",
       "FL     54.73  \n",
       "IL  No Value  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add Columns\n",
    "df2['year'] = [2019,2019,2020,2019,2020]\n",
    "df2['month'] = [2,5,7,9,11]\n",
    "df2['day'] = [27,12,10,7,17]\n",
    "df2['gdp'] = ['3183', '1918', '1751', '1111', '908']\n",
    "df2['rainfall'] = ['22.9',35,42.87,'54.73','No Value']\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Checking Data Types\n",
    "Using the `dtypes` property of a `DataFrame`, we can check the different Data Types of each column in a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "population    float64\n",
       "land area     float64\n",
       "density       float64\n",
       "drought        object\n",
       "year            int64\n",
       "month           int64\n",
       "day             int64\n",
       "gdp            object\n",
       "rainfall       object\n",
       "dtype: object"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__`info()`__ function can also be used to get a more detailed information; however, it is a more __time consuming__ operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 5 entries, CA to IL\n",
      "Data columns (total 9 columns):\n",
      " #   Column      Non-Null Count  Dtype  \n",
      "---  ------      --------------  -----  \n",
      " 0   population  5 non-null      float64\n",
      " 1   land area   5 non-null      float64\n",
      " 2   density     5 non-null      float64\n",
      " 3   drought     5 non-null      object \n",
      " 4   year        5 non-null      int64  \n",
      " 5   month       5 non-null      int64  \n",
      " 6   day         5 non-null      int64  \n",
      " 7   gdp         5 non-null      object \n",
      " 8   rainfall    5 non-null      object \n",
      "dtypes: float64(3), int64(3), object(3)\n",
      "memory usage: 560.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "df2.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the data types of different columns, we can identify a few issues:\n",
    "\n",
    "1. `year`, `month` and `day` columns are of type `int64` and should be converted to `datetime64`.\n",
    "2. `gdp` is of type `object` and should be `float64`.\n",
    "3. `drought`, a Yes/No column is of type `object` and should be converted to `bool`.\n",
    "4. `rainfall` is also of type `object` and should be converted to `float64`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Changing Data Types\n",
    "\n",
    "In __Pandas__, there are three options to convert the data type of a column:\n",
    "\n",
    "1. Using `astype()` function to to force the appropriate data type.\n",
    "2. Writing a custom function.\n",
    "3. Using pandas built-in functions such as `to_numeric()` and `to_datetime()`.\n",
    "\n",
    "Let's look at each of these options."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Using `astype()` function\n",
    "`astype()` is the simplest way to convert the data type. Let's start by converting the `gdp` column of type `object` to `float64` data type. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA    3183.0\n",
       "TX    1918.0\n",
       "NY    1751.0\n",
       "FL    1111.0\n",
       "IL     908.0\n",
       "Name: gdp, dtype: float64"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['gdp'].astype('float')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "population    float64\n",
       "land area     float64\n",
       "density       float64\n",
       "drought        object\n",
       "year            int64\n",
       "month           int64\n",
       "day             int64\n",
       "gdp            object\n",
       "rainfall       object\n",
       "dtype: object"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simply running `astype()` on a column only returns a copy of the column. To ensure the change gets applied to the DataFrame, we need to assign it back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Change data type\n",
    "df2['gdp'] = df2['gdp'].astype('float')\n",
    "\n",
    "# Check data type\n",
    "df2['gdp'].dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try changing the data types of `drought` (dtype object) to `bool` and `rainfall` (dtype object) to `float64`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Change `drought` to `bool` type__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA    True\n",
       "TX    True\n",
       "NY    True\n",
       "FL    True\n",
       "IL    True\n",
       "Name: drought, dtype: bool"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['drought'].astype('bool')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Here we see that after type conversion, all values are interpreted as `True`, however, the column consisted of both `Yes` and `No` values. By default, if there is a value in column, python evaluates it to `True` irrespective of the type of value. Learn more about truth value evaluation [here](https://docs.python.org/3/library/stdtypes.html#truth)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Change `rainfall` to `float64` type__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> This results in a `ValueError` which means the conversion of `rainfall` to `float64` data type did not work.\n",
    "\n",
    "`astype()` works when the data is clean and when you want to convert a numeric to a string object. However, when the data is not homogeneous (i.e. `object`) or has characters, `astype` may not be the right way to change the data type of a column.\n",
    "\n",
    "Let's look at the other options of converting data types (mentioned above) to see if we can fix these issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Custom Functions\n",
    "\n",
    "We can write custom functions and then apply them to convert the data type of a column to an appropriate type. Let's change the data type of `drought` (dtype object) to `bool`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create function \n",
    "def str2bool(val):\n",
    "    \"\"\"\n",
    "    Converts a string data type to boolean\n",
    "    - converts string values to lowercase\n",
    "    - checks for multiple positive values \n",
    "    - returns the correct boolean type\n",
    "    \"\"\"\n",
    "    \n",
    "    val = val.lower() # converts string value to lowercase\n",
    "    if val in (\"yes\", \"y\", \"true\",  \"t\", \"1\"): # checks for multiple positive values\n",
    "        return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the `apply` function to apply the `str2bool` function to all values of the column. \n",
    "\n",
    "Remember that simply applying a function to a column only returns a copy of the column. To ensure the change gets applied to the DataFrame, we need to assign it back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('bool')"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply the function\n",
    "df2['drought'] = df2['drought'].apply(str2bool)\n",
    "\n",
    "# Check type\n",
    "df2['drought'].dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA     True\n",
       "TX     True\n",
       "NY    False\n",
       "FL     True\n",
       "IL    False\n",
       "Name: drought, dtype: bool"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Look at column values\n",
    "df2['drought']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The beauty of custom functions is that they open up a gateway of opportunities. With a simple function, we could consider multiple string values such as `\"yes\", \"y\", \"true\", \"t\", \"1\"`. As a result, the column can essentially have any of these values and the function will return `True`. \n",
    "\n",
    "This was just a simple function and much more complex functions can be written to address various issues in the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Using `np.where()`\n",
    "\n",
    "Another popular approach that is useful for many types of problems is to use `np.where()` function. Let's convert the data type of `drought` back to `object` and then take a look at using `np.where()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('O')"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['drought'] = ['Y','Y','N','Y','N']\n",
    "df2['drought'].dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use np.where()\n",
    "df2['drought'] = np.where(df2['drought'] == 'Y', True, False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA     True\n",
       "TX     True\n",
       "NY    False\n",
       "FL     True\n",
       "IL    False\n",
       "Name: drought, dtype: bool"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check values\n",
    "df2['drought']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`np.where()` function converts all`Y` values to `True` and everything else is changed to `False`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Pandas built-in functions\n",
    "\n",
    "Pandas built-in helper functions, such as `to_numeric()` and `to_datetime()`, can be very useful for converting certain data type. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Convert `rainfall` column (type object) to `float`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA        22.9\n",
       "TX          35\n",
       "NY       42.87\n",
       "FL       54.73\n",
       "IL    No Value\n",
       "Name: rainfall, dtype: object"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Look at rainfall column\n",
    "df2['rainfall']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA      <class 'str'>\n",
       "TX      <class 'int'>\n",
       "NY    <class 'float'>\n",
       "FL      <class 'str'>\n",
       "IL      <class 'str'>\n",
       "Name: rainfall, dtype: object"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the data type of each value in the column\n",
    "df2['rainfall'].apply(type)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `rainfall` column contains values of multiple different types, such as integers, floats and strings. Let's use the `to_numeric` function to change data type to `float64`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA    22.90\n",
       "TX    35.00\n",
       "NY    42.87\n",
       "FL    54.73\n",
       "IL      NaN\n",
       "Name: rainfall, dtype: float64"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.to_numeric(df2['rainfall'], errors='coerce')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By passing `errors=coerce`, the function replaces the invalid “No Value” value with a `NaN`.\n",
    "\n",
    "To ensure the change gets applied to the DataFrame, we need to assign it back. Here is another way of applying `to_numeric` using the `apply` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['rainfall'] = df2['rainfall'].apply(pd.to_numeric, errors='coerce')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA    22.90\n",
       "TX    35.00\n",
       "NY    42.87\n",
       "FL    54.73\n",
       "IL      NaN\n",
       "Name: rainfall, dtype: float64"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['rainfall']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Convert `year, month, day` columns (type int) to `datetime64`\n",
    "\n",
    "The `year, month, day` columns can be combined into a single new `date` column with the correct data type. Let's convert the date columns to `datetime64` type using `pd.to_datetime()`. We will dive into the details of individual date or time columns in a later section of this guide series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2['date'] = pd.to_datetime(df2[['year','month','day']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CA   2019-02-27\n",
       "TX   2019-05-12\n",
       "NY   2020-07-10\n",
       "FL   2019-09-07\n",
       "IL   2020-11-17\n",
       "Name: date, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the data types of `df2` DataFrame are all cleaned up. Let's take a look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "population           float64\n",
       "land area            float64\n",
       "density              float64\n",
       "drought                 bool\n",
       "year                   int64\n",
       "month                  int64\n",
       "day                    int64\n",
       "gdp                  float64\n",
       "rainfall             float64\n",
       "date          datetime64[ns]\n",
       "dtype: object"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operating on DataFrames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Pandas__ is designed to work with __NumPy__ and essentially inherits the ability to perform quick element-wise operations with basic arithmetic (add, subtract, multiply, etc.) and more sophisticated operations (trigonometric, exponential and logarithmic functions, etc.) from __NumPy__.\n",
    "\n",
    "There are various other ways in which users can interact with the data in a __DataFrame__, such as reindexing data, dropping data, adding data, renaming columns etc. Let's explore how we can operate on the data in a DataFrame.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Arithmetic Operations\n",
    "In __Pandas__, \n",
    "- Unary ufuncs (operate on a single input): such as exponential and logarithmic functions, preserve index and column labels in the output.\n",
    "- Binary ufuncs (operate on two inputs): such as addition and multiplication, automatically align indices and return a DataFrame whose index and columns are the unions of the ones in each DataFrame.\n",
    "\n",
    "Let's create sample data to demonstrate this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df3 is: \n",
      "            b  c   d\n",
      "Ohio      14  8   9\n",
      "Texas      6  6   2\n",
      "Colorado  17  4  15\n",
      "\n",
      "df4 is: \n",
      "           b   d   e\n",
      "Arizona  14  13  16\n",
      "Ohio      8   2  22\n",
      "Texas    24   9  20\n",
      "Iowa     11   8   1\n"
     ]
    }
   ],
   "source": [
    "# Create data\n",
    "rng = np.random.RandomState(55)\n",
    "\n",
    "df3 = pd.DataFrame(rng.randint(1,20,(3,3)), columns=list('bcd'),\n",
    "                    index=['Ohio', 'Texas', 'Colorado'])\n",
    "print('df3 is: \\n', df3)\n",
    "\n",
    "print()\n",
    "\n",
    "df4 = pd.DataFrame(rng.randint(1,25,(4,3)), columns=list('bde'),\n",
    "                   index=['Arizona', 'Ohio', 'Texas', 'Iowa'])\n",
    "print('df4 is: \\n', df4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Unary Operations\n",
    "\n",
    "When applying a NumPy ufunc on DataFrame object, the result will be a Pandas object with the indices preserved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exponentiation:\n",
      "             b   c    d\n",
      "Ohio      196  64   81\n",
      "Texas      36  36    4\n",
      "Colorado  289  16  225\n",
      "\n",
      "Square Root:\n",
      "                  b         c         d\n",
      "Ohio      3.741657  2.828427  3.000000\n",
      "Texas     2.449490  2.449490  1.414214\n",
      "Colorado  4.123106  2.000000  3.872983\n",
      "\n",
      "ln(x)\n",
      "                 b         d         e\n",
      "Arizona  2.639057  2.564949  2.772589\n",
      "Ohio     2.079442  0.693147  3.091042\n",
      "Texas    3.178054  2.197225  2.995732\n",
      "Iowa     2.397895  2.079442  0.000000\n"
     ]
    }
   ],
   "source": [
    "print('Exponentiation:\\n', np.power(df3,2))\n",
    "print()\n",
    "print('Square Root:\\n', np.sqrt(df3))\n",
    "print()\n",
    "print(\"ln(x)\\n\", np.log(df4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Binary Operations\n",
    "\n",
    "Pandas will automatically align indices and return a DataFrame whose index and columns are the unions of the ones in each DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b   c     d   e\n",
       "Arizona    NaN NaN   NaN NaN\n",
       "Colorado   NaN NaN   NaN NaN\n",
       "Iowa       NaN NaN   NaN NaN\n",
       "Ohio      22.0 NaN  11.0 NaN\n",
       "Texas     30.0 NaN  11.0 NaN"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Addition\n",
    "df3 + df4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the `c` and `e` columns are not found in both DataFrame objects, they appear as all missing in the result. The same holds for the rows whose labels are not common in both DataFrame.\n",
    "\n",
    "We can also use __Pandas Methods__ to perform arithmetic operations. Adding these DataFrame will result in __NA__ values in the locations that don’t overlap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b   c     d   e\n",
       "Arizona    NaN NaN   NaN NaN\n",
       "Colorado   NaN NaN   NaN NaN\n",
       "Iowa       NaN NaN   NaN NaN\n",
       "Ohio      22.0 NaN  11.0 NaN\n",
       "Texas     30.0 NaN  11.0 NaN"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Addition\n",
    "df3.add(df4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>-18.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-7.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b   c    d   e\n",
       "Arizona    NaN NaN  NaN NaN\n",
       "Colorado   NaN NaN  NaN NaN\n",
       "Iowa       NaN NaN  NaN NaN\n",
       "Ohio       6.0 NaN  7.0 NaN\n",
       "Texas    -18.0 NaN -7.0 NaN"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Subtraction\n",
    "df3 - df4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>112.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>144.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              b   c     d   e\n",
       "Arizona     NaN NaN   NaN NaN\n",
       "Colorado    NaN NaN   NaN NaN\n",
       "Iowa        NaN NaN   NaN NaN\n",
       "Ohio      112.0 NaN  18.0 NaN\n",
       "Texas     144.0 NaN  18.0 NaN"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Multiplication\n",
    "df3 * df4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>1.75</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>0.25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b   c         d   e\n",
       "Arizona    NaN NaN       NaN NaN\n",
       "Colorado   NaN NaN       NaN NaN\n",
       "Iowa       NaN NaN       NaN NaN\n",
       "Ohio      1.75 NaN  4.500000 NaN\n",
       "Texas     0.25 NaN  0.222222 NaN"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Division\n",
    "df3/df4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Operations between DataFrame and Series\n",
    "\n",
    "Operations between a DataFrame and a Series are similar to the operations between a two-dimensional and one-dimensional NumPy array. The index and column alignment is maintained when applying arithmetic operations between a DataFrame and a Series. \n",
    "\n",
    "As an example, when we subtract a two-dimensional array and one of its rows, the subtraction is performed once for each row based on NumPy's broadcasting rules. Operations between a DataFrame and a Series are similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>17</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           b  c   d\n",
       "Ohio      14  8   9\n",
       "Texas      6  6   2\n",
       "Colorado  17  4  15"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get Data\n",
    "df3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>-8</td>\n",
       "      <td>-2</td>\n",
       "      <td>-7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>3</td>\n",
       "      <td>-4</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          b  c  d\n",
       "Ohio      0  0  0\n",
       "Texas    -8 -2 -7\n",
       "Colorado  3 -4  6"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Subtract row for `Ohio` from df3\n",
    "df3 - df3.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting DataFrame shows element values when the row for `Ohio` gets subtracted from the DataFrame.\n",
    "\n",
    "For operating on columns, you can use the object methods while specifying the `axis` keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Colorado</th>\n",
       "      <th>Ohio</th>\n",
       "      <th>Texas</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Colorado  Ohio  Texas   b   c   d\n",
       "Ohio           NaN   NaN    NaN NaN NaN NaN\n",
       "Texas          NaN   NaN    NaN NaN NaN NaN\n",
       "Colorado       NaN   NaN    NaN NaN NaN NaN"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df3.subtract(df3['c'], axis='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The resulting DataFrame shows element values when column `c` gets subtracted from the DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reindexing Data\n",
    "`reindex` is an important Pandas method that conforms the data to a new index. `reindex`, when applied to a `DataFrame`, can alter either the (row) index, columns, or both. Lets' take a look at reindexing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CA</th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CA  TX  OH\n",
       "a   0   1   2\n",
       "c   3   4   5\n",
       "e   6   7   8"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create data\n",
    "df5 = pd.DataFrame(np.arange(9).reshape((3, 3)),\n",
    "                    index=['a', 'c', 'e'],\n",
    "                    columns=['CA', 'TX', 'OH'])\n",
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Reindexing Rows__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CA</th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    CA   TX   OH\n",
       "e  6.0  7.0  8.0\n",
       "d  NaN  NaN  NaN\n",
       "c  3.0  4.0  5.0\n",
       "b  NaN  NaN  NaN\n",
       "a  0.0  1.0  2.0"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5 = df5.reindex(['e','d','c','b','a'])\n",
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The index for this DataFrame is now updated. For the new indices `b and d` that were added with reindexing, Pandas has automatically generated `Nan` values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Reindexing Columns__\n",
    "\n",
    "Columns can be reindexed using the `columns` keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CA</th>\n",
       "      <th>OH</th>\n",
       "      <th>TX</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>6.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    CA   OH   TX\n",
       "e  6.0  8.0  7.0\n",
       "d  NaN  NaN  NaN\n",
       "c  3.0  5.0  4.0\n",
       "b  NaN  NaN  NaN\n",
       "a  0.0  2.0  1.0"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5 = df5.reindex(columns=['CA','OH','TX'])\n",
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Reindexing Rows and Columns__\n",
    "\n",
    "`reindex` can be used more succinctly by label-indexing with `loc`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "      <th>CA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TX   OH   CA\n",
       "a  1.0  2.0  0.0\n",
       "b  NaN  NaN  NaN\n",
       "c  4.0  5.0  3.0\n",
       "d  NaN  NaN  NaN\n",
       "e  7.0  8.0  6.0"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5 = df5.loc[['a','b','c','d','e'], ['TX','OH','CA']]\n",
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Reset Index\n",
    "`reset_index()` can be used to reset the index of a DataFrame to a default index. This operations turns the index labels into columns and assigns a default numerical index to the DataFrame.  To modify the DataFrame use `inplace=True`.\n",
    "\n",
    "`reset_index()` comes in handy when working with data that has multiple levels of indexing. We will dig into the details of `MultiIndex` objects in the next part of this guide series. For now, let's take a quick look at how it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "      <th>CA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>d</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>e</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  index   TX   OH   CA\n",
       "0     a  1.0  2.0  0.0\n",
       "1     b  NaN  NaN  NaN\n",
       "2     c  4.0  5.0  3.0\n",
       "3     d  NaN  NaN  NaN\n",
       "4     e  7.0  8.0  6.0"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`drop` parameter can be used to avoid the old index being added as a column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "      <th>CA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TX   OH   CA\n",
       "0  1.0  2.0  0.0\n",
       "1  NaN  NaN  NaN\n",
       "2  4.0  5.0  3.0\n",
       "3  NaN  NaN  NaN\n",
       "4  7.0  8.0  6.0"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dropping Data\n",
    "Dropping rows or columns comes in handy when cleaning your data. The `drop()` function can be easily applied. \n",
    "\n",
    "Functions that modify the size or shape of a DataFrame return a new object so that the original data remains unchanged. To apply changes to existing DataFrame, we need to either assign the function back to DataFrame or use `inplace` keyword."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Dropping rows__\n",
    "\n",
    "Calling drop with index labels will drop values from the row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "      <th>CA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TX   OH   CA\n",
       "a  1.0  2.0  0.0\n",
       "c  4.0  5.0  3.0\n",
       "e  7.0  8.0  6.0"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop rows\n",
    "df5.drop(['b','d'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of `drop` operation is a new object as it does not modify the original DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "      <th>CA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TX   OH   CA\n",
       "a  1.0  2.0  0.0\n",
       "b  NaN  NaN  NaN\n",
       "c  4.0  5.0  3.0\n",
       "d  NaN  NaN  NaN\n",
       "e  7.0  8.0  6.0"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To modify the DataFrame, we will use `inplace` keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "      <th>CA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TX   OH   CA\n",
       "a  1.0  2.0  0.0\n",
       "c  4.0  5.0  3.0\n",
       "e  7.0  8.0  6.0"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5.drop(['b','d'], inplace=True)\n",
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Dropping Columns__\n",
    "\n",
    "Columns can be dropped by passing a value to the `axis` keyword: `axis=1` or `axis='columns'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TX   OH\n",
       "a  1.0  2.0\n",
       "c  4.0  5.0\n",
       "e  7.0  8.0"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5.drop('CA', axis='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adding Data\n",
    "A key aspect of data exploration is to feature engineer the data i.e. add new features to the data. New columns can be easily added to a DataFrame using the following methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Using Values in a List__\n",
    "\n",
    "A new column can be added with default values from a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "      <th>CA</th>\n",
       "      <th>FL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TX   OH   CA  FL\n",
       "a  1.0  2.0  0.0   5\n",
       "c  4.0  5.0  3.0   8\n",
       "e  7.0  8.0  6.0  10"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5['FL'] = [5,8,10]\n",
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column for `FL` is now added to the end of the DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Using `insert()`__\n",
    "\n",
    "`insert()` provides the flexibility to add a column at any index position. It also provides different options for inserting the column values. Let's add a new column for `NY` at index position `2` between `OH` and `CA`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "      <th>NY</th>\n",
       "      <th>CA</th>\n",
       "      <th>FL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9</td>\n",
       "      <td>6.0</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TX   OH  NY   CA  FL\n",
       "a  1.0  2.0   7  0.0   5\n",
       "c  4.0  5.0   4  3.0   8\n",
       "e  7.0  8.0   9  6.0  10"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5.insert(loc=2, column='NY', value=[7,4,9])\n",
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Using `assign()`__\n",
    "\n",
    "`assign()` returns a new DataFrame object that has a copy of the original data with the requested changes. The original DataFrame remains unchanged. To apply changes to existing DataFrame, we need to assign the function back to the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "df5 = df5.assign(WA = [0,3,12])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TX</th>\n",
       "      <th>OH</th>\n",
       "      <th>NY</th>\n",
       "      <th>CA</th>\n",
       "      <th>FL</th>\n",
       "      <th>WA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9</td>\n",
       "      <td>6.0</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TX   OH  NY   CA  FL  WA\n",
       "a  1.0  2.0   7  0.0   5   0\n",
       "c  4.0  5.0   4  3.0   8   3\n",
       "e  7.0  8.0   9  6.0  10  12"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Renaming Columns\n",
    "Real world data is messy. In your data exploration journey, you may come across column names that are not representative of the data or that are too long, or you may just want to standardize the names of columns in your dataset. Let's look at how we can rename columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Using `rename()`__\n",
    "\n",
    "`rename()` is quite useful when we need to rename some selected columns. Let's rename some columns to reflect the names of states."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "df5.rename(columns = {'TX':'Texas', 'OH':'Ohio'}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Texas</th>\n",
       "      <th>Ohio</th>\n",
       "      <th>NY</th>\n",
       "      <th>CA</th>\n",
       "      <th>FL</th>\n",
       "      <th>WA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9</td>\n",
       "      <td>6.0</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Texas  Ohio  NY   CA  FL  WA\n",
       "a    1.0   2.0   7  0.0   5   0\n",
       "c    4.0   5.0   4  3.0   8   3\n",
       "e    7.0   8.0   9  6.0  10  12"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__By assigning a list of Columns__\n",
    "\n",
    "We can also assign a list of new names to the `columns` attribute of the DataFrame object. The caveat is that this method required _providing new names for all the columns_ even if want to rename only a few."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "df5.columns = ['Texas', 'Ohio', 'New York', 'California', 'Florida', 'Washington']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Texas</th>\n",
       "      <th>Ohio</th>\n",
       "      <th>New York</th>\n",
       "      <th>California</th>\n",
       "      <th>Florida</th>\n",
       "      <th>Washington</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>9</td>\n",
       "      <td>6.0</td>\n",
       "      <td>10</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Texas  Ohio  New York  California  Florida  Washington\n",
       "a    1.0   2.0         7         0.0        5           0\n",
       "c    4.0   5.0         4         3.0        8           3\n",
       "e    7.0   8.0         9         6.0       10          12"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with Missing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missing data occurs in many applications as real world data in rarely clean. It is generally referred to as _Null, NaN, or NA values_. Assessing the missing data is an important process, as it can identify potential problems or biases that may arise as a result of the missing data. There are various useful methods for detecting, removing, and replacing null values in __Pandas__ such as:\n",
    "- `isnull()`: generates a boolean mask indicating missing values.\n",
    "- `notnull()`: generates a boolean mask indicating non-missing values.\n",
    "- `dropna()`: drops null values and returns a filtered version of the data.\n",
    "- `fillna()`: returns a copy of the data with missing values filled or imputed.\n",
    "- `interpolate()`: powerful function that providers various interpolation techniques to fill the missing values.\n",
    "\n",
    "Let's start by looking at the types of missing data in Pandas and then we will explore how to detect, filter, drop and impute missing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Missing Data in Pandas\n",
    "\n",
    "__Pandas__ uses two already existing Python null values:\n",
    "\n",
    "- `None` - is a Python singleton object only used in arrays with data type 'object'.\n",
    "- `NaN` - is a special floating point value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### `None` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, None, 4, 6], dtype=object)"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v1 = np.array([1, None, 4, 6])\n",
    "v1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`dtype=object` shows NumPy inferred that the contents of this array are Python objects. Any operations on the data will be done at the Python level, which are typically slower than the  arrays with native types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dtype = object\n",
      "391 µs ± 5.76 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n",
      "\n",
      "dtype = int\n",
      "11.2 µs ± 708 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n",
      "\n",
      "dtype = float\n",
      "11.2 µs ± 653 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for dtype in ['object', 'int', 'float']:\n",
    "    print(\"dtype =\", dtype)\n",
    "    %timeit np.arange(1E4, dtype=dtype).sum()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Aggregation_ operations on an array with `None` value results in an error. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### `NaN`: Not a Number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v2 = np.array([1, np.NaN, 4, 6])\n",
    "v2.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`dtype('float64')` shows NumPy inferred that the contents of this array are native floating-point type. Unlike `object`, this array supports faster operations.\n",
    "\n",
    "_Aggregation_ operations on an array with `NaN` will result in a `NaN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(nan, nan)"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v2.sum(), v2.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### `NaN` and `None`\n",
    "__Pandas__ handles both `NaN` and `None` interchangeably and automatically converts them as appropriate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.0\n",
       "1    NaN\n",
       "2    5.0\n",
       "3    NaN\n",
       "dtype: float64"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series([1, np.nan, 5, None])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas automatically converts `None` to a `NaN` value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Detect Missing Data\n",
    "\n",
    "`isnull()` and `notnull()` methods can be used to detect and filter missing data. We will start by creating some data to apply these methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa       NaN   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create data\n",
    "df6 = df3 + df4\n",
    "df6.loc['Ohio','c'] = 10\n",
    "df6.loc['Colorado','b'] = 5\n",
    "df6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              b      c      d     e\n",
       "Arizona    True   True   True  True\n",
       "Colorado  False   True   True  True\n",
       "Iowa       True   True   True  True\n",
       "Ohio      False  False  False  True\n",
       "Texas     False   True  False  True"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.isnull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result shows a matrix of True or False values based on the presence of null values in data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Missing values in each column\n",
    "\n",
    "Chaining a `sum()` method returns a summation of missing values in each column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b    2\n",
       "c    4\n",
       "d    3\n",
       "e    5\n",
       "dtype: int64"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Missing values in DataFrame\n",
    "Chaining two `sum()` methods will return the total number of missing values in the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.isnull().sum().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Using `notnull()`\n",
    "`notnull()` is the opposite of `isnull()` and can be used to check the number of non-missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b    3\n",
       "c    1\n",
       "d    2\n",
       "e    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.notnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Total non-missing values in dataframe\n",
    "df6.notnull().sum().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Visualize Missing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to understand the pattern of occurrence of missing values before we decide to drop or impute missing data. An easy way to visualize missing records is to use `heatmap()` from the seaborn library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Seaborn\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa       NaN   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get Data\n",
    "df6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fb2e87b5a90>"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAD4CAYAAADCb7BPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAOq0lEQVR4nO3de7BddXmH8eebhAoRDBepAVEoHQKCNcEkKDWximVoO7USKgK1KmrJH62l0TJOK7ZSWztO7UXrtZERlFpxqsZLy4gYaxJtNSYhCQlIrKgVEWoBwUChGN7+sVfaYzhJTn7nsnJyns9MhnXWXmuvd+0hefjtncNJVSFJ0r6a1vcAkqTJyYBIkpoYEElSEwMiSWpiQCRJTWb0PcBEevTOOf6VM0naB9Nmb8tuH5vIQSRJBw4DIklqYkAkSU0MiCSpiQGRJDUxIJKkJgZEktTEgEiSmhgQSVITAyJJamJAJElNDIgkqYkBkSQ1MSCSpCYGRJLUxIBIkpoYEElSEwMiSWoy6oAkWZKkkpyyh2P+dbTXkSTtX8ZiBXIR8CXgwl0fSDIdoKp+fgyuI0naj4wqIEkOBZ4DvJouIEmel+RfkvwDcFO3b3v3zzcn2dj9+l6Sq7r9r0uypfu1rNt3QpJbkrw/ydYkn0tySPfYJUm+lmRTko8nmTma+5Ak7bvRrkDOBT5bVduAe5I8s9t/BnB5VZ069OCq+uOqmgf8AnA38K4k84FXAs8Cng1ckuT07pSTgHdX1WnAD4Ff7/Z/oqoWVtVc4BYGARtWkqVJ1iVZt/ya+0Z5u5KknUYbkIuAa7vta7uvAdZW1beGOyFJgA8Df1NV64FFwIqqeqCqtgOfABZ3h3+rqjZ22+uBE7rtpydZk+Qm4KXAabsbsKqWV9WCqlqw9GWzmm5SkvRYM1pPTHIUcBaDP8wLmA4UcB3wwB5OvQK4vaqu2vlUezj24SHbO4BDuu2rgXOralOSi4Hn7eP4kqRRGs0K5MXAh6rq+Ko6oaqeAnyLwYpiWEl+FTgbuHTI7tXAuUlmJnk8sARYs5drHwZ8P8lBDFYgkqQJNpqAXASs2GXfx4Hf2MM5vw8cC6ztPkh/c1VtYLCiWAt8Fbiyqm7cy7X/qDv2BuDrDbNLkkYpVdX3DBPm0TvnTJ2blaQxMG32tt1+zOB3okuSmhgQSVITAyJJamJAJElNDIgkqYkBkSQ1MSCSpCYGRJLUxIBIkpoYEElSEwMiSWpiQCRJTQyIJKmJAZEkNTEgkqQmzT/SdjI659i5fY9wwLj+jk19jyCpZ65AJElNDIgkqYkBkSQ1MSCSpCYGRJLUxIBIkpoYEElSEwMiSWpiQCRJTQyIJKmJAZEkNTEgkqQmBkSS1MSASJKaGBBJUhMDIklqYkAkSU0MiCSpyYgCkmR2kmuTfDPJzUmuSzJnN8eekGTL2I75f899RZLLxuO5JUn7Zq8BSRJgBfDFqvrZqjoVeAPwpLEYIMmU+rnsknSgGMkK5PnAI1X1vp07qmoj8KUkb0uyJclNSS7Y9cQkBye5qnv8xiTP7/ZfnOQfk3wG+FySQ5OsTLKhO/ZFQ57j8iS3Jvk8cPKQ/fOSfCXJ5iQrkhwxitdBkrSPRvJf/08H1g+z/zxgHjAXeCLwtSSrdznmdwCq6ueSnMIgFjvf+joTeEZV3dOtQpZU1f1Jngh8JcmngWcCFwKnd7NuGDLLh4DfrapVSd4MvAlYNqK7liSN2mg+RF8EfKSqdlTVXcAqYOEwx1wDUFVfB74D7AzIDVV1T7cd4M+TbAY+DzyZwVtki4EVVfVgVd0PfBogySzg8Kpa1Z3/QeC5ww2ZZGmSdUnW3V63jeJ2JUlDjSQgW4H5w+zPCM7d0zEPDNl+KXA0ML+q5gF3AQd3j9UIrrNbVbW8qhZU1YLjcuJonkqSNMRIAvIF4HFJLtm5I8lC4F7ggiTTkxzNYAWwdpdzVzOIA91bV08Fbh3mGrOA/6yqR7rPSY4fcv6SJIckOQx4IUBV3Qfcm2Rxd9zLGKyAJEkTZK+fgVRVJVkCvD3JHwAPAd9m8HnDocAmBquE11fVnUlOGHL6e4D3JbkJ+DFwcVU9PPiLXT/hw8BnkqwDNgJf7669IclHu33fAdYMOecV3XPPBG4DXrkP9y1JGqVUjeodoknl7GnnT52bHWfX37Gp7xEkTYBps7ft9qMIvxNdktTEgEiSmhgQSVITAyJJamJAJElNDIgkqYkBkSQ1MSCSpCYGRJLUxIBIkpoYEElSEwMiSWpiQCRJTQyIJKmJAZEkNTEgkqQmU+oHSj1655ypc7OSNAb8gVKSpDFnQCRJTQyIJKmJAZEkNTEgkqQmBkSS1MSASJKaGBBJUhMDIklqYkAkSU0MiCSpiQGRJDUxIJKkJgZEktTEgEiSmhgQSVITAyJJamJAJElNeglIku19XFeSNHZcgUiSmvQakAy8LcmWJDcluaDb/54kv9Ztr0jygW771Un+rNv+ZJL1SbYmWdrfXUjS1NT3CuQ8YB4wF/hF4G1JjgFWA4u7Y54MnNptLwLWdNuvqqr5wALg0iRHDXeBJEuTrEuybvk1943TbUjS1DOj5+svAj5SVTuAu5KsAhYyiMSyJKcCNwNHdGE5E7i0O/fSJEu67acAJwF373qBqloOLAd49M45NZ43I0lTSd8ByXA7q+p7SY4AfonBauRI4CXA9qr6UZLnMVixnFlVDyb5InDwxIwsSYL+38JaDVyQZHqSo4HnAmu7x/4NWNYdswa4jP9/+2oWcG8Xj1OAZ0/s2JKkvgOyAtgMbAK+ALy+qu7sHlsDzKiqfwc2MFiF7AzIZ4EZSTYDfwp8ZUKnliSRqqnzsYCfgUjSvpk2e9uwHzVA/ysQSdIkZUAkSU0MiCSpiQGRJDUxIJKkJgZEktTEgEiSmhgQSVITAyJJamJAJElNDIgkqYkBkSQ1MSCSpCYGRJLUxIBIkpr0/SNtJ9Q5x87tewRJ4+z6Ozb1PcKU4QpEktTEgEiSmhgQSVITAyJJamJAJElNDIgkqYkBkSQ1MSCSpCYGRJLUxIBIkpoYEElSEwMiSWpiQCRJTQyIJKmJAZEkNTEgkqQmBkSS1MSASJKa9BaQJMcl+VSSbyT5ZpJ3JPmpJBcnedduzrkuyeETPask6bF6CUiSAJ8APllVJwFzgEOBt+zpvKr6lar64QSMKEnai75WIGcBD1XVVQBVtQN4LfAqYCZwbJLPdquTv9h5UpJvJ3lit/26JFu6X8t6uAdJmtJm9HTd04D1Q3dU1f1J/oPBTPOA04GHgVuTvLOqvrvz2CTzgVcCzwICfDXJqqq6caJuQJKmur5WIAFqD/tXVtV9VfUQcDNw/C7HLQJWVNUDVbWdwdthi4e9ULI0ybok626v28buDiRpiusrIFuBBUN3JHkC8BRgB4OVx047eOxKKSO9UFUtr6oFVbXguJzYOK4kaVd9BWQlMDPJywGSTAf+CrgaeHAE568Gzk0yM8njgSXAmnGaVZI0jF4CUlXF4A/985N8A9gGPAS8YYTnb2AQm7XAV4Er/fxDkiZWBn+WTw1nTzt/6tysNEVdf8emvkc4oEybvW23Hxn4neiSpCYGRJLUxIBIkpoYEElSEwMiSWpiQCRJTQyIJKmJAZEkNTEgkqQmBkSS1MSASJKaGBBJUhMDIklqYkAkSU0MiCSpiQGRJDXZ9WeNSyPiD+0ZO+ccO7fvEaQmrkAkSU0MiCSpiQGRJDUxIJKkJgZEktTEgEiSmhgQSVITAyJJamJAJElNDIgkqYkBkSQ1MSCSpCYGRJLUxIBIkpoYEElSEwMiSWpiQCRJTQyIJKnJuP9I2yRHASu7L2cDO4AfdF+fUVX/M94zSJLG3rgHpKruBuYBJLkC2F5Vfzne15Ukja9e3sJKMj/JqiTrk1yf5Jgks5LcmuTk7piPJLmk235vknVJtib5kyHP89YkNyfZnMQoSdIEGvcVyDACvBN4UVX9IMkFwFuq6lVJXgNcneQdwBFV9f7unMur6p4k04GVSZ4B3A4sAU6pqkpy+LAXS5YCSwFO4ZkclxPH+fYkaWroIyCPA54O3JAEYDrwfYCquiHJ+cC7gblDznlJF4IZwDHAqcDNwEPAlUn+Gfin4S5WVcuB5QBnTzu/xuOGJGkq6msFsrWqznzMA8k04GnAfwNHArcn+RngMmBhVd2b5Grg4Kr6cZIzgBcAFwKvAc6aoHuQpCmvj89AHgaOTnImQJKDkpzWPfZa4BbgIuADSQ4CngA8ANyX5EnAL3fnHQrMqqrrgGV0H9RLkiZGHyuQR4EXA3+bZFY3w9uTPAL8FoO/2vujJKuBN1bVm5LcCGwFbgO+3D3PYcCnkhzMYFXz2om+EUmayiY0IFV1xZAvnzvMIU8bcuzrhmxfvJunPGNMBpMk7TO/E12S1MSASJKaGBBJUhMDIklqYkAkSU0MiCSpiQGRJDUxIJKkJgZEktTEgEiSmhgQSVITAyJJamJAJElNDIgkqYkBkSQ1MSCSpCapqr5n0BBJllbV8r7nOFD4eo4tX8+xNdlfT1cg+5+lfQ9wgPH1HFu+nmNrUr+eBkSS1MSASJKaGJD9z6R9P3Q/5es5tnw9x9akfj39EF2S1MQViCSpiQGRJDUxIPuJJCck2dL3HNLeJLkiyWV9z6H+GRBJUhMDsn+ZkeSDSTYn+ViSmX0PNJkleXn3Wm5Kck3f80xmSS5PcmuSzwMn9z3PZJfkN5OsTbIxyd8lmd73TC0MyP7lZGB5VT0DuB/47Z7nmbSSnAZcDpxVVXOB3+t5pEkryXzgQuB04DxgYb8TTW5JngZcADynquYBO4CX9jtVGwOyf/luVX252/57YFGfw0xyZwEfq6r/Aqiqe3qeZzJbDKyoqger6n7g030PNMm9AJgPfC3Jxu7rE/sdqc2MvgfQT9j1m3L8Jp12wddvLPlajp0AH6yqP+x7kNFyBbJ/eWqSM7vti4Av9TnMJLcSeEmSowCSHNnzPJPZamBJkkOSHAa8sO+BJrmVwIuT/DQM/t1McnzPMzUxIPuXW4BXJNkMHAm8t+d5Jq2q2gq8BViVZBPw1z2PNGlV1Qbgo8BG4OPAmn4nmtyq6mbgjcDnut/rNwDH9DtVG/9XJpKkJq5AJElNDIgkqYkBkSQ1MSCSpCYGRJLUxIBIkpoYEElSk/8FklA/Bd7hyuoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize missing\n",
    "sns.heatmap(df6.isnull(), cbar=False, cmap='viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missing records are displayed in __yellow__ color."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Filter based on Missing Data\n",
    "\n",
    "Both `isnull()` and `notnull()` can be applied to columns of a DataFrame to filter out rows with missing and non-missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa       NaN   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get data\n",
    "df6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter rows where values in column `b` are null."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          b   c   d   e\n",
       "Arizona NaN NaN NaN NaN\n",
       "Iowa    NaN NaN NaN NaN"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6[df6['b'].isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filter rows where values in column `b` are not null."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6[df6['b'].notnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Drop Missing Data\n",
    "\n",
    "`dropna()` method allows us to drop any missing values from the data. The method returns a copy of the DataFrame, so to apply the changes inplace, use `inplace=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa       NaN   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get data\n",
    "df6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `dropna()` will drop all rows in which any null value is present."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [b, c, d, e]\n",
       "Index: []"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since all rows of `df6` have some NA values, the result is an empty copy of the DataFrame. Alternatively, using `axis='columns'` drops all columns containing a null value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: [Arizona, Colorado, Iowa, Ohio, Texas]"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.dropna(axis='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since all columns have some NA values, the result is an empty copy of the DataFrame.\n",
    "\n",
    "`dropna()` provides the flexibility to drop rows or columns with all NA values, or a majority of NA values using __`how`__ or __`thresh`__ parameters. The default `how='any'`, allows any row or column containing a null value to be dropped. To only drop rows or columns that have all null values, `how='all'` can be specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop rows with all null values\n",
    "df6.dropna(how='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d\n",
       "Arizona    NaN   NaN   NaN\n",
       "Colorado   5.0   NaN   NaN\n",
       "Iowa       NaN   NaN   NaN\n",
       "Ohio      22.0  10.0  11.0\n",
       "Texas     30.0   NaN  11.0"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop columns with all null values\n",
    "df6.dropna(how='all', axis='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__`thresh`__ parameter allows you specify a minimum number of non-null values a row/column should have to be kept in the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          b     c     d   e\n",
       "Ohio   22.0  10.0  11.0 NaN\n",
       "Texas  30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.dropna(thresh=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only rows with 2 or more non-null values are kept, and since the row for `Colorado` has only 1 non-null value, it is dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>d</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     d\n",
       "Arizona    NaN   NaN\n",
       "Colorado   5.0   NaN\n",
       "Iowa       NaN   NaN\n",
       "Ohio      22.0  11.0\n",
       "Texas     30.0  11.0"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.dropna(thresh=2, axis='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, column `c` has only 1 non-null value and is therefore dropped."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Impute Missing Data\n",
    "\n",
    "Rather than dropping NA values and potentially discarding some other data with it, you may just want to replace them with a value such as 0, or some other imputation such as mean or median of the data. Pandas `fillna()` method can be used for such operations. The method returns a new object, but you can modify the existing object in-place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa       NaN   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get data\n",
    "df6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Fill with a Constant Value\n",
    "\n",
    "Calling `fillna()` with a constant value replaces missing values with that value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d    e\n",
       "Arizona    0.0   0.0   0.0  0.0\n",
       "Colorado   5.0   0.0   0.0  0.0\n",
       "Iowa       0.0   0.0   0.0  0.0\n",
       "Ohio      22.0  10.0  11.0  0.0\n",
       "Texas     30.0   0.0  11.0  0.0"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Forawrd and Backward Fill\n",
    "\n",
    "Forward and backward fill can be used to propagate the previous value forward (`ffill`) or next values backward (`bfill`). We can specify an `axis` along which the fill method will operate.\n",
    "\n",
    "__Note__: If a previous value is not available during a fill operation, the NA value remains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa       5.0   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0  10.0  11.0 NaN"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Forward fill along rows\n",
    "df6.fillna(method='ffill')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>11.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d     e\n",
       "Arizona    NaN   NaN   NaN   NaN\n",
       "Colorado   5.0   5.0   5.0   5.0\n",
       "Iowa       NaN   NaN   NaN   NaN\n",
       "Ohio      22.0  10.0  11.0  11.0\n",
       "Texas     30.0  30.0  11.0  11.0"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Forward fill along columns\n",
    "df6.fillna(method='ffill', axis='columns')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    5.0  10.0  11.0 NaN\n",
       "Colorado   5.0  10.0  11.0 NaN\n",
       "Iowa      22.0  10.0  11.0 NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Backward fill along rows\n",
    "df6.fillna(method='bfill')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Fill with Arithmetic or Aggregate Functions\n",
    "\n",
    "A mean, median, mode, max or min value for the column can be used to fill missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa       NaN   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get data\n",
    "df6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>19.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>19.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona   19.0  10.0  11.0 NaN\n",
       "Colorado   5.0  10.0  11.0 NaN\n",
       "Iowa      19.0  10.0  11.0 NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0  10.0  11.0 NaN"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fill with mean\n",
    "df6.fillna(df6.mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mohi9282/opt/anaconda3/envs/arcgis-clone/lib/python3.8/site-packages/numpy/lib/nanfunctions.py:1115: RuntimeWarning: All-NaN slice encountered\n",
      "  r, k = function_base._ureduce(a, func=_nanmedian, axis=axis, out=out,\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona   22.0  10.0  11.0 NaN\n",
       "Colorado   5.0  10.0  11.0 NaN\n",
       "Iowa      22.0  10.0  11.0 NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0  10.0  11.0 NaN"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fill with median\n",
    "df6.fillna(df6.median())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>30.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>30.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona   30.0  10.0  11.0 NaN\n",
       "Colorado   5.0  10.0  11.0 NaN\n",
       "Iowa      30.0  10.0  11.0 NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0  10.0  11.0 NaN"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fill with max\n",
    "df6.fillna(df6.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Fill using a Dictionary of Values\n",
    "\n",
    "A dictionary of constant values or aggregate functions can be passed to fill missing values in columns differently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa       NaN   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>22.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>22.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d    e\n",
       "Arizona   22.0   NaN  11.0  1.0\n",
       "Colorado   5.0   NaN  11.0  1.0\n",
       "Iowa      22.0   NaN  11.0  1.0\n",
       "Ohio      22.0  10.0  11.0  1.0\n",
       "Texas     30.0   NaN  11.0  1.0"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fill_value = {'b': df6['b'].median(),\n",
    "              'd': df6['d'].mean(),\n",
    "              'e': 1}\n",
    "df6.fillna(value=fill_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Using `interpolate()`\n",
    "\n",
    "`interpolate()` is a very powerful function that providers various interpolation techniques (linear, quadratic, polynomial etc.) to fill the missing values. Let's take a quick look, and you can learn more about `interpolate()` [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.interpolate.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa       NaN   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0   NaN  11.0 NaN"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get data\n",
    "df6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`interpolate()` using Linear method. Note that Linear method ignores the index and treats the values as equally spaced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Colorado</th>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Iowa</th>\n",
       "      <td>13.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ohio</th>\n",
       "      <td>22.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Texas</th>\n",
       "      <td>30.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             b     c     d   e\n",
       "Arizona    NaN   NaN   NaN NaN\n",
       "Colorado   5.0   NaN   NaN NaN\n",
       "Iowa      13.5   NaN   NaN NaN\n",
       "Ohio      22.0  10.0  11.0 NaN\n",
       "Texas     30.0  10.0  11.0 NaN"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df6.interpolate(method='linear', limit_direction='forward')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part of the guide series we introduced __[Pandas](https://pandas.pydata.org/)__, a Python package that builds on __[NumPy](https://numpy.org/)__ and provides data structures and functions designed to make working with structured data fast, easy, and expressive. \n",
    "\n",
    "You have seen how `DataFrame` can be created and then data can be accessed using `loc` and `iloc` operators. We discussed in detail how to check the different data types in a DataFrame and ways to change these data types. We also discussed how to perform various operations on a `DataFrame` (i.e. Arithmetic, Reindex, Add and Drop data) and to work with missing data. We briefly introduced working with a `Series` object as well.\n",
    "\n",
    "In the next part of this guide series, you will learn about how to be more productive with Pandas. We will discuss data aggregation, transformation operations, multi-level indexing and working with time series data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] Wes McKinney. 2017. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython (2nd. ed.). O'Reilly Media, Inc.    \n",
    "    \n",
    "[2] Jake VanderPlas. 2016. Python Data Science Handbook: Essential Tools for Working with Data (1st. ed.). O'Reilly Media, Inc.    \n",
    "\n",
    "[3] https://pbpython.com/pandas_dtypes.html"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  },
  "livereveal": {
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "275px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
